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QUANTIZATION  AND  SOURCE  ENCODING  WITH  A 
FIDELITY  CRITERION:  A SURVEY 


INTRODUCTION 

In  information  theory,  it  is  generally  known  that  characterizations  of  the  optimum 
source  encoder  and  source  code  for  a given  source  are  extremely  difficult  to  obtain.  In 
fact,  a large  part  of  information  theory  is  devoted  to  this  and  similar  problems.  Evalua- 
tion of  the  optimum  encoding  performance  is  also  difficult,  but  more  progress  has  been 
made  in  this  area.  Such  evaluation  is  usually  based  on  a theoretically  derived  optimum 
performance  curve,  called  the  rate-distortion  function,  which  is  obtainable  without  knowl- 
edge of  the  optimal  encoder  description.  The  solutions  to  these  two  companion  problems 
have  been  the  goal  of  many  information  theorists. 

Optimal  source  encoding  for  an  entire  class  of  information  sources  is  considerably 
more  difficult  and  complex.  In  this  situation,  the  evaluation  of  the  best  possible  encoding 
performance  has  been  carried  out  in  only  a few  instances,  and  characterization  of  the  opti- 
mal source  encoder  has  not  been  achieved.  Implementation  of  optimal  source  encoders 
for  classes  of  sources,  it  is  hoped,  will  not  be  too  complex. 

For  these  reasons,  quantizers  have  been  considered  for  encoding  discrete  time  infor- 
mation sources  and  classes  of  these  sources  because  of  their  simple  implementation,  their 
familiarity  to  engineers  and  the  volume  of  associated  research  results.  In  this  survey  we 
provide  a brief  analysis  of  an  exhaustive  list  of  references  up  through  1975,  which  has 
been  categorized  according  to  the  significant  research  and  development  issues  in  the  areas 
of  quantization,  source  encoding,  and  their  interrelationship.  The  goal  is  to  provide  an 
information  base  for  further  research  and  development. 

The  next  section  discusses  the  relationship  between  source  encoding  with  a fidelity 
criterion  and  quantization;  it  thus  serves  as  a prelude  to  the  survey.  The  literature  survey 
is  divided  into  two  topic  areas,  source  encoding  and  quantization.  A brief  summary  and 
conclusion  constitute  Sec.  4. 

The  author  apologizes  to  those  whose  work  has  been  omitted. 


RELATIONSHIP  BETWEEN  SOURCE  ENCODING  WITH  A 
FIDELITY  CRITERION  (RATE-DISTORTION  THEORY) 

AND  QUANTIZATION 

Source  encoding  with  a fidelity  criterion,  as  introduced  by  Shannon  [52],  or  rate- 
distortion  theory  [6J,  as  it  is  now  called,  is  a major  area  of  information  theory  [1-63] 
and  is  also  the  mathematical  basis  for  data  compression  [6,14],  Rate-distortion  theory 
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is  concerned  with  specification  of  the  noiseless  communication  channel  or  noiseless  memory 
capacity  for  an  acceptable  reproduction  error  or  distortion  level  at  the  receiver  or  user. 

The  rate-distortion  function  is  the  basis  for  this  specification  and  the  foundation  of  the 
theory.  Hence,  knowledge  of  the  rate-distortion  function  is  of  considerable  benefit  to 
the  communication  system  designer,  and  the  determination  of  rate-distortion  functions  for 
various  sources  and  classes  of  sources  has  been  an  active  area  of  research. 

The  rate-distortion  function  R(d)  for  a given  information  or  signal  source  and  a given 
distortion  measure,  specifies  the  minimum  channel  capacity  or  the  minimum  source  infor- 
mation required  for  reproducing  the  entire  source  output  at  the  receiver  with  an  average 
distortion  of  d or  less  (i.e.,  to  achieve  fidelity  d).  This  function  is  computed  by  minimi- 
zing the  mutual  information,  between  the  source  and  the  channel  output,  over  all  possible 
channels  that  generate  a distortion  level  of  d or  less.  In  general,  this  computation  uses 
variational  techniques  [6,11,33]  and  is  not  easy.  For  a distortion  measure  that  assigns 
d = 0 (for  perfect  reproduction  of  the  input  at  the  output),  R(0)  is  just  the  entropy  of 
the  source.  R(d)  decreases  monotonically  with  d and  usually  becomes  zero  at  some  finite 
value  of  distortion,  say  dmax,  indicating  that  no  information  need  be  transmitted  to 
achieve  a distortion  value  of  dmax. 

An  equally  active  research  area  is  the  search  for  source-encoding  techniques  to  achieve 
the  rate-distortion  function  values  for  given  distortion  levels.  Early  research  in  this  area 
dealt  almost  exclusively  with  the  search  for  optimal  source-encoding  techniques  yielding 
performance  improvements  that  would  justify  their  cost  and  added  complexity.  This 
research  was  fruitful  only  in  some  areas,  e.g.,  space  communications  and  high-density 
data  storage  [59].  Many  source-encoding  theorists  recently  have  been  engaged  in  finding 
good  suboptimal  source-encoding  techniques  that  are  easy  to  implement  [1,2,13,35,40, 
56,59,63] . 

The  main  objective  of  using  a source-encoding  technique  such  as  quantization  is  to 
achieve  an  acceptable  balance  between  the  distortion  in  reproducing  the  input  and  the 
cost  of  acquiring,  transmitting,  and/or  storing  the  additional  data  needed  for  perfect 
reproduction  of  the  input.  Amplitude  quantization  of  source  signals  is  probably  the 
first  source-encoding  technique,  and  it  is  easy  to  implement  [80,90,93,98,111],  e.g.,  in 
analog-to-digital  conversion.  In  fact,  quantization  can  be  considered  the  cornerstone  of 
sampled-data  theory  and  systems  [68,83,90,98,101,110,112],  Considerable  research  has 
been  conducted  on  complete  or  partial  source  encoding  of  stochastic  sources  by  quanti- 
zation [9,40,64,66,68,70,72,73,75,82,84,87,92,94,97,99,100,102,109,111,112,114,116]. 

Quantization  is  a nonlinear,  zero-memory  operation  of  transforming  a source  variable, 
possibly  random  and  possibly  having  a continuum  of  values,  into  an  output  variable 
having  a finite  number  (N)  of  values.  The  device  performing  this  transformation  is  called 
an  N-level  quantizer,  and  its  output  is  an  N-step  approximation  to  its  input.  Distortion 
results  from  trying  to  reproduce  the  input  from  the  output.  A fidelity  criterion  (for 
example  a particular  value  of  d)  is  assigned  to  provide  a quantitative  measure  of  the 
approximation  accuracy  of  reproduction.  The  most  common  fidelity  criterion  is  a bound 
on  the  mean  weighted  quantization  error,  where  this  error  is  just  the  nonnegatively 
weighted  difference  between  the  quantizer  input  and  output. 
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Optimal  quantization  of  a given  source  is  a scheme  that  optimizes  the  distortion 
measure  for  that  particular  source.  In  general,  such  quantizers  perform  poorly  with 
respect  to  other  sources  or  distortion  measures.  Robust  quantization,  of  a class  of 
sources,  is  a quantization  scheme  that  quarantees  a performance  level  or  fidelity  criterion 
for  all  sources  in  that  class  [40,95,111,112].  In  general,  robust  quantization  is  sub- 
optimal  for  most  sources  in  the  class. 


LITERATURE  SURVEY 
Source  Encoding 

C.  E.  Shannon  presented  the  information  theory  in  1949  [51]  and  the  rate-distortion 
theory  in  1959  [52].  Berger’s  book  [6]  is  the  only  one  devoted  exclusively  to  rate- 
distortion  theory.  He  includes  an  extensive  bibliography  and  covers  most  previous 
research  results.  Other  books,  such  as  those  of  Gallager  [21]  and  Sakrison  [49],  have  at 
least  one  chapter  covering  the  basics  of  this  theory.  Published  papers  devoted  to  rate- 
distortion  theory  per  se  are  those  of  Berger  [3,5],  Berger  and  Yu  [8],  Davisson  and 
Gray  [16],  Davisson  and  Pursley  [17],  Goblick  [24],  Gray  [32],  Gray  and  Davisson 
[28-30],  Gray  and  Wyner  [31],  Kieffer  [36],  Leiner  [37],  Leiner  and  Gray  [38],  Rubin 
[42],  Wolf  and  Ziv  [58],  Wyner  [62],  and  Ziv  [63,  Part  II].  Results  of  the  evaluation 
and/or  bounding  of  rate-distortion  (distortion-rate)  functions  (also  the  e-entropy  of 
Kolmogorov)  were  reported  by  many  researchers,  including  Berger  [3],  Binia  et  al.  [10], 
Blahut  [12],  Dobrushin  [19,20],  Gray  [25-27],  Haskell  [33],  Marton  [39],  Morris  [40], 
Rubin  [42,43],  Sakrison  [47,48],  Tan  and  Yao  [55],  and  Wyner  and  Ziv  [60], 

Although  memoryless  source  models  represent  many  meaningful  applications  it  has 
generally  been  acknowledged  that  substantial  benefits  from  source  coding  can  be  achieved 
for  sources  with  memory  [3,4,6,17,25-27,35,53].  Likewise,  distortion  measures  incor- 
porating symbol  context  dependences  [8]  or  the  concept  of  source  coding  when  side 
information  is  available  to  the  encoder-decoder  pair  [32,37,38,62]  have  been  found  to  be 
applications  with  substantial  coding  gain  potential. 

The  investigation  of  source-encoding  techniques  that  achieve  the  performance  prom- 
ised by  rate-distortion  theory  has  been  an  active  area  of  research.  This  investigation 
parallels  the  earlier  research  for  channel  encoding  techniques  to  achieve  the  performance 
promised  by  the  channel  coding  theorems  of  information  theory.  In  the  same  manner, 
this  newer  area  of  research  is  a major  interest  of  coding  theorists  (e.g.,  Berger,  Jelinek, 
and  Wolf  [7],  Jelinek  [34],  Jelinek  and  S-  > .euler  [35],  Slepian  and  Wolf  [53],  Viterbi 
and  Omura  157],  Wolf  |59),  and  Ziv  |63]).  Many  others  have  reported  results  in  this 
area  [1,2, 13, 18,22, 12, 32J . Implementation  of  the  various  encoding  techniques  is  gener- 
ally not  simple,  although  current  large-scale  integration  technology  is  making  implemen- 
tation complex  easier  to  handle.  An  exception  is  source  encoding  by  quantization.  For 
example,  Berger  [9]  demonstrated  an  equivalence  between  perfectly  entropy-encoded 
optimum  quantizers  and  infinite  block-length  permutation  codes,  Chang  [12]  expanded 
and  implemented  Berger’s  results,  and  Morris  [40]  showed  that  the  uniform  quantizer 
generates  an  adequate  rate-distortion  bound  (within  1 or  2 bits  per  second  of  Gaussian 
curve)  for  a class  of  sources  that  includes  the  Gaussian. 
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Classes  of  Sources 

Published  results  on  source  encoding  for  a class  of  sources  are  few,  and  most  of 
these  results  are  recent.  This  problem  has  been  treated  by  Berger  [5,6),  Davisson  [15], 
Dobrushin  [19,20],  Morris  [40],  Neuhoff  [41],  Sakrison  [47,48,50],  Tan  and  Yao  [55], 
and  Ziv  [63],  Berger  used  a two-person  statistical  game  approach,  with  respect  to  a 
distortion  measure,  for  a class  of  sources  and  a class  of  encoding  schemes.  He  determined 
the  exponential  rate  of  growth  (with  block  length)  of  the  minimum  number  of  code 
words  needed  to  achieve  a specified  distortion  level.  Dobrushin  gave  an  expression  for 
the  e-entropy  (a  point  on  the  rate-distortion  function)  of  a class  of  sources.  In  both  of 
these  investigations,  the  class  of  sources  were  limited  to  discrete-time,  discrete-amplitude 
sources.  Morris  theoretically  verified  the  robustness  (minimax)  of  the  uniform  quantizer 
for  a class  of  discrete-time,  memoryless  sources  with  bounded  amplitudes  and  then  used 
this  result  to  derive  an  upper  bound  on  the  rate-distortion  function  for  that  class  of 
sources.  This  bound  is  achieved  by  the  uniform  quantizers.  Sakrison,  in  two  pioneering 
papers  [47,48] , extended  Dobrushin ’s  results  to  continuous-amplitude,  discrete-time 
sources  and  to  random  processes.  He  stated  that  for  a “compact”  class  of  sources,  the 
rate-distortion  function  for  the  class  of  sources  is  just  the  supremum  of  the  rate-distortion 
functions  of  the  individual  sources  and  hence  is  more  easily  computed.  In  Ref.  50, 
Sakrison  investigated  robust  encoding  schemes  and  worst-case  sources.  He  showed  that 
for  a class  of  ergodic  sources  satisfying  a moment  equality  constraint,  the  Gaussian  source 
is  the  worst-case  source.  He  then  conjectured  that  an  optimum  encoding  for  the  worst- 
case  source  is  the  desired  robust  encoder.  Tan  and  Yao  presented  a method  of  explicitly 
evaluating  an  absolute-magnitude-criterion,  rate-distortion  function  for  a class  of  inde- 
pendent and  identically  distributed  sources  having  probability  densities  with  constrained 
tail  decay.  Berger,  Morris,  Sakrison,  and  Tan,  and  Yao  appear  to  be  the  only  investi- 
gators making  contributions  in  this  area  for  the  last  5 to  10  years. 

Davisson  [15],  Neuhoff  et  al.  [41],  and  Ziv  [63],  among  others,  have  treated 
another  form  of  source  encoding,  for  a class  of  stationary  sources,  called  universal  coding. 
An  encoder  is  called  universal  if  the  performance  of  the  code,  designed  with  only  the 
knowledge  that  the  true  source  is  a member  of  a given  class  of  sources,  converges  in  the 
limit  of  long  block  length  to  the  optimum  performance  possible  if  one  knew  the  true 
source.  Different  types  of  universality,  which  correspond  to  different  notions  of  con- 
vergence, are  defined.  Davisson  considered  universal  noiseless  coding;  i.e.,  perfect  repro- 
duction of  the  source,  where  the  performance  measure  is  a function  of  the  coding  redun- 
dancy relative  to  the  per-letter  conditional  source  entropy.  He  also  gives  a short  history 
of  universal  coding.  Neuhoff  et  al.  investigated  universal  coding  with  a fidelity  criterion 
with  the  added  constraint  that  the  admissible  codes  have  a given  fixed  code  rate.  The 
performance  measure  was  some  function  of  the  distortion  vs  the  minimum  information 
rate.  Ziv  pioneered  universal  coding  with  a fidelity  criterion  [63,  Part  II].  He  also  con- 
sidered universal  coding  for  the  performance  measure,  probability  of  coding  error,  using 
a fixed  rate  [63,  Part  I]. 

Implementation  of  these  encoding  techniques  for  classes  of  sources  is  generally 
difficult  (e.g.,  Gilbert  [23] ),  with  the  exception  of  quantization.  In  fact,  for  the  problem 
formulations  treated  in  Refs.  19,  20,  47,  and  50  the  encoding  techniques  for  achieving 
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the  rate-distortion  function  (e-entropy  in  Refs.  19  and  20)  performance,  and  the  charac- 
terization of  the  worst-case  source  of  the  class,  are  not  known.  One  exception  seems  to 
be  for  the  class  of  sources  with  a given  bound  on  the  variances  or  with  a given  variance 
(47,50].  In  these  two  cases,  the  Gaussian  source  is  the  worst-case  source  [51] . However, 
the  form  of  the  corresponding  optimal  encoding  technique  for  the  class  of  sources  (i.e., 
the  robust  encoder)  has  only  been  conjectured  [50].  For  this  reason,  the  robust  quanti- 
zation results  of  Morris  [40],  Morris  and  VandeLinde  [95],  and  VandeLinde  [111,112] 
have  practical  significance  for  memoryless  sources. 

Noise-Corrupted  Sources 

Source  encoding  of  noise-corrupted  source  signals  has  had  limited  attention  from 
rate-distortion  theorists  [44,49],  Assuming  source-independent  random  noise  and  a mean- 
squared  distortion  measure,  Sakrison  [44]  and  Wolf  and  Ziv  [49]  formulated  the  problem 
in  Hilbert  space.  Sakrison  needed  sufficiently  long  word  lengths  (i.e.,  large  dimensionality) 
to  take  advantage  of  the  assumed  ergodicity  of  the  source.  Wolf  and  Ziv’s  results  were 
applicable  to  any  work  lengths.  They  concluded  [44,49]  that  the  optimal  encoding 
technique  is  first  to  compute  the  optimum  conditional  estimate  of  the  source  and  then 
to  encode  this  estimate  optimally  as  if  it  were  the  source  output.  They  derived  expres- 
sions for  the  minimum  possible  distortion  measure  value,  whose  computational  difficulty 
is  generally  greater  than  for  the  noiseless  case.  The  problem  of  characterizing  and  imple- 
menting the  optimal  encoder  still  exists,  since  the  source  estimator  is  now  the  “source” 
to  be  optimally  encoded. 

Quantization 

W.  F.  Sheppard,  according  to  Bruce  [70]  and  Smith  [108] , was  the  first  to  study 
a system  of  quantization.  Sheppard,  in  1898,  essentially  performed  a round-off  operation 
before  making  computations  on  the  entries  of  a table  of  values  [107],  The  most  refer- 
enced research  work  on  quantization  are  probably  the  papers  of  J.  Max  [92]  and  B. 
Widrow  [114].  Max  considered  the  problem  of  the  optimum  N-level  quantizer  for  the 
minimum  mean  weighted  quantization  error  for  a given  signal.  He  also  considered  the 
optimum  N-level  uniform  quantizer  (i.e.,  equally  spaced  quantizer  level  values  and  equally 
spaced  transition  values).  For  the  mean-squared  error  and  sufficiently  smooth  signal 
density  problem,  Max  showed  that  the  centroids  of  the  signal  density  area  in  each  quan- 
tization interval  are  the  optimum  quantizer  level  values,  and  the  midpoints  between  the 
optimum  level  values  are  the  optimum  transition  points.  The  actual  values  must  be 
computed  recursively  from  the  resulting  necessary  condition  equations.  Explicit  results 
were  given  for  the  Gaussian  source  code.  Widrow  provided  theoretical  verification  of  the 
earlier  experimental  results  of  Bennett  [66]  by  showing  that  the  instantaneous  quanti- 
zation error,  which  is  a signal-dependent  error  signal,  can  be  considered  statistically  inde- 
pendent uniformly  distributed  noise  when  the  number  of  quantization  levels  N satisfies 
a condition  analagous  to  the  Nyquist  sampling  theorem.  He  considered  the  uniform 
quantization  operation  as  an  area-sampling  operation  on  the  signal  probability  density 
function.  When  quantization  frequency  N is  twice  as  high  as  the  highest  frequency 
component  in  the  signal  density  function,  the  moments  of  the  quantizer  output  signal 
are  the  same  as  the  moments  of  the  sum  of  the  quantizer  input  signal  and  a statistically 
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independent  noise  uniformly  distributed  in  a quantization  interval.  Satisfaction  of  this 
condition,  according  to  his  quantizing  theorem,  also  allows  complete  recovery  of  the 
original  probability  density,  given  the  quantizer  output  probability  density. 

Other  Performance  Measures 

Most  of  the  research  on  quantization  schemes  considered  the  mean  weighted  quanti- 
zation error  as  the  distortion  measure,  with  the  mean  rth  power  quantization  error  the 
most  common  of  that  group.  Other  measures  considered  were  based  on  special  applica- 
tions or  information  theoretic  viewpoints  that  reflected  the  level  of  knowledge  in  the 
respective  application  areas.  Signal-to-noise  ratios  were  used  as  the  distortion/performance 
measure  for  digitizing  speech  and  picture  signals  in  Refs.  73,  81,  82,  84,  102,  and  108. 

The  mean  rth  power  of  the  quantization  interval  length,  within  which  the  signal  is  found, 
was  used  for  determining  bounds  on  quantizer  performance  Ref.  77.  The  quantizer  output 
entropy  was  used  as  a basis  for  comparison  of  quantizer  performance  from  an  information 
theoretic  viewpoint  in  Refs.  79,  80,  and  94.  An  overall  system  performance  measure  may 
be  used  in  some  applications  where  the  quantizer  is  considered  just  another  part  of  a 
system.  The  overall  system  performance  was  the  optimization  goal  in  Refs.  68,  89,  90, 
and  110.  Purton  [102]  and  Smith  [108]  considered  zero-memory  prefiltering  and  post- 
filtering (called  companding)  of  the  quantizer  input  and  output,  to  equalize  S/N  over  the 
typical  range  of  the  speaker  volume  means.  Their  papers,  especially  Ref.  108,  are  prob- 
ably the  best  treatments  of  companding.  Cummiskey  et  al.  [24] , Gersho  and  Goodman 
[78],  Golding  and  Schultheiss  [81],  Goodman  and  Gersho  [82],  and  Jayant  [87]  in- 
vestigated adaptive  quantization  to  achieve  the  same  equalization  of  S/N  ratios.  In  this 
formulation,  the  interval  size  of  the  uniform  quantizer  is  adjusted  for  the  next  signal 
value,  which  is  predicted  from  knowledge  of  the  previous  values.  Elias  [77]  derived  lower 
bounds  on  his  distortion  measure  for  input  signal  density  constraints  of  absolute  contin- 
uity and  finite  domain.  He  also  presented  a good  literature  survey  and  an  extensive 
bibliography.  Messerschmitt  [94]  showed  that  the  maximum-output-entropy  quantizer 
and  the  minimum  mean  rth-power  quantization -error  quantizer  are  approximately  the 
same,  within  a multiplicative  constant,  for  a maximum-output-entropy  goal.  He  required 
the  source  signal  densities  to  be  uniform  or  exponential  in  form.  Goblick  and  Holsinger 
[80],  in  an  important  paper,  used  curves  of  the  output  entropy  vs  mean-squared  quanti- 
zation error  as  a basis  for  comparison  of  Gaussian  source  digitization  schemes.  Their 
often  quoted  result  is  that  the  uniform  quantizer  (Max’s  optimal  quantizer)  performance 
curve  is  1/4  bit  above  the  rate-distortion  function  (the  theoretical  limit)  for  the  Gaussian 
source.  Gish  and  Pierce  [79]  presented  an  asymptotic  (in  N)  comparison  between  the 
uniform  quantizer  and  the  rate-distortion  function  for  mean  rth-power  quantization-error 
distortion  measure  and  reasonably  smooth  signal  densities.  They  showed  that  the  uniform 
quantizer  yields  an  output  entropy  asymptotically  smaller  than  that  of  any  other  quantizer, 
for  reasonably-smooth  signal  density  functions  and  a fairly  general  class  of  mean  weighted 
quantization  error  distortion  measures.  Larson  [89],  Lewis  and  Tou  [90],  and  Tou  [110] 
used  the  system  performance  index  or  measure  and  considered  the  quantizer  as  just  a 
small  part  of  a dynamic  sampled-data  system.  Bertram  [68]  obtained  an  expression  for 
the  upper  bound  of  the  error  caused  by  quantization  in  such  systems.  Viswanathan  and 
Marhoul  [113]  used  a spectral  sensitivity  measure  on  a desired  spectrum  shape  as  the 
basis  of  quantizer  optimization  for  application  to  digitally  transmitted,  linear  predictive 
system  parameters. 
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Optimal  quantization  with  respect  to  the  mean  weighted-quantization-error  distortion 
measure  was  investigated  by  many  researchers  after  the  paper  by  Max,  the  primary  moti- 
vation being  the  interest  in  digital  techniques,  which  began  in  the  early  1960s.  Bruce 
[69,70]  developed  a dynamic  programing  algorithm  to  specify  the  quantizer  parameters 
that  minimize  this  distortion  measure  (Ref.  70  has  an  extensive  bibliography).  The 
quantization-error  weighting  function  in  this  formulation  is  not  assumed  to  be  symmetric 
or  convex.  However,  unless  the  weighting  function  is  convex,  one  must  search  for  many 
possible  relative  extrema  to  obtain  the  optimum  parameters.  Roe  [103]  extended  Max’s 
results  to  mean  rth-power  quantization-error  distortion  measures  and  to  signal  density 
functions  that  are  sufficiently  differentiable  to  allow  Taylor  series  expansions  about  the 
transition  points.  Algazi  [64]  and  Wood  [115]  presented  other  approximation  techniques 
for  reducing  the  computational  complexity  and  size  of  the  quantization-parameter  optimi- 
zation algorithms  given  in  previous  papers  [69,70,93,103].  The  results  of  Algazi  and 
Wood  are  based  on  large  N and  on  sufficiently  smooth  signal  density  functions.  Morris 
[40],  Morris  and  VandeLinde  [95],  and  VandeLinde  [111,112]  used  a functional  analysis 
formulation  to  verify  the  robustness  of  certain  N-level  quantizers  in  a minimax  sense  for 
the  classes  of  sources. 

Random  Vector  Quantization 

Quantization  of  random  vectors  has  been  reported  by  only  a few  researchers.  This 
type  of  formulation  is  especially  important  to  information  theorists  because  of  their 
interest  in  encloding  of  signal  source  sequences  (often  called  n-dimensional  message,  or 
block  encoding),  which  is  characteristic  of  multiplexed  systems.  Schutzenberger  [107], 
in  a relatively  early  paper,  investigated  the  reciprocal  relationship  between  the  mean  rth- 
power  quantization  error  and  the  quantizer  output  entropy  for  the  quantization  of  finite 
dimensional  signals.  He  gave  a lower  bound  to  the  distortion  measure,  which  is  propor- 
tional to  N_r/n.  His  results  require  continuous,  bounded  signal  density  functions  that 
have  a finite  0 moment  for  some  0 > r and  have  a finite  entropy.  The  constant  of 
proportionality  was  not  explicitly  presented.  Huang  [85]  and  Huang  and  Schultheiss 
[86]  treated  the  quantization  of  random  Gaussian  vectors.  They  determined  a matrix 
transformation  that  converted  the  correlated  elements  of  each  vector  to  uncorrelated 
elements  and  then  quantized  each  element  in  an  optimal  manner.  Zador  [116],  in  a 
later,  unpublished  work,  gave  an  asymptotic  result  that  is  similar  to  that  of  Ref.  105. 
Berger’s  [9]  work  on  optimum  quantizer  and  permutation  code  equivalence  was  formu- 
lated for  random  sequences.  Segall  [106]  treated  the  problem  of  optimum  allocation 
of  the  total  number  of  available  bits  (quantization  levels)  among  the  components  of  a 
memoryless,  stationary  vector  source.  He  gave  an  optimal  decorrelating  scheme  for  a 
source  with  dependent  vector  components. 

Performance  Experiments 

Research  concerned  with  the  results  of  performance  experiments  on  specific  quan- 
tizers has  been  a very  important  field  of  quantization  research.  Bennett  [66]  was  one 
of  the  first  at  this  type  of  research.  He  demonstrated  in  his  timely  and  important  paper 
that  the  quantization-noise  (error)  power  spectral  density  was  uniformly  distributed  over 
the  input  signal  band.  He  assumed  a uniform  quantizer  with  “more  than  a few  number 
of  intervals”  and  a Gaussian  source  with  a flat  band-limited  power  spectral  density.  After 
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analyzing  his  results,  he  suggested  tapering  the  quantizer  interval  lengths  so  that  the  weak 
speech  signals  occurred  in  the  shorter  intervals  and  the  strong  speech  signals  in  the  longer 
intervals.  This  is  the  fundamental  idea  behind  companding  [102,108],  as  discussed  earlier. 
O’Neal  [99,100]  investigated  the  performance  of  differential  pulse  code  modulation  and 
delta  modulation  systems  for  transmission  of  Gaussian  and  television  signals.  This  method 
of  encoding  is  often  called  predictive  quantization,  since  only  the  quantized  difference 
between  the  current  signal  value  and  the  predicted  value  is  transmitted.  He  also  looked 
at  the  resulting  quantization  error.  Damas  et  al.  [76]  and  Thompson  and  Sparkes  [109] 
experimentally  investigated  the  effects  of  additive  random  noise,  called  dithering,  on  the 
signal  for  the  purpose  of  whitening  the  quantization  noise.  This  procedure  seems  to  be 
very  effective  in  digital  picture  processing.  Arnstein  [65]  treated  the  quantization  error 
in  predictive  coders  for  first-order  Gaussian  Markov  sequences.  His  results  indicate  that 
the  prediction  error  is  close  to  Gaussian  and  that  correlation  of  successive  quantizer  outputs 
does  exists. 

Sensitivity  Analysis 

The  application  of  sensitivity  analysis  to  quantizer  optimization  or  performance 
evaluation  has  not  been  treated  in  the  literature  with  any  depth.  A lack  of  an  adequate 
sensitivity  measure  may  have  been  the  major  inhibition  to  earlier  use  of  sensitivity  analysis. 
Viswanathan  and  Marhoul  [115]  were  concerned  with  an  accurate  representation  of  the 
power  spectrum  for  synthesizing  good-quality  speech.  Thus,  a spectral  sensitivity  measure 
was  the  basis  for  their  quantizer  optimization  of  the  linear  prediction  parameter.  Gray 
and  Davisson  [85]  applied  a generalization  of  the  Vasershtein  distance  between  source 
random  variables  to  derive  an  upper  bound  on  the  difference  in  quantizer  performance 
for  different  sources.  Their  bound  provides  a measure  of  the  performance  loss  or  mismatch 
that  occurs  in  applying  a quantizer  to  a source  for  which  it  was  not  designed. 

Noise-Corrupted  Sources 

All  previously  referenced  research  treated  the  ideal  case  of  noiseless  input  signals  to 
the  quantizer.  Only  a few  investigators  treated  the  quantization  of  noise-corrupted  signal 
sources.  In  1956,  Myers  [28]  considered  the  uniform  quantization  of  uniformly  distri- 
buted sources  with  additive  random  Gaussian  noise.  He  showed  that  for  small-variance 
noise,  the  quantizer  output  would  be  essentially  the  same  as  the  noiseless  case,  and  an 
increased  number  of  quantization  intervals  would  yield  good  improvement  in  the  total 
error.  For  large  noise  variances,  the  quantizer  output  would  be  incorrect  a large  percentage 
of  the  time,  and  an  increased  number  of  quantization  intervals  would  yield  very  little 
improvement  in  the  total  error  (i.e.,  the  noise  would  dominate  the  total  error).  Myers 
suggested  the  probability  of  a correct  reading  as  a measure  of  quantizer  performance  or 
as  a measure  of  allowable  noise  limits.  Myers  appears  to  have  been  the  first  to  have 
treated  this  type  of  problem.  Bruce  [70]  extended  this  work  and  others  by  determining 
an  algorithm  for  computing  the  optimum  quantizer  parameters  that  minimize  the  mean 
weighted-quantization-error  distortion  measure  for  noise-corrupted  signal  sources.  The 
corrupting  noise  need  not  be  additive.  The  other  problem  constraints  are  the  same  as  his 
noiseless  quantization  problem.  For  the  mean  squared-error  case,  Bruce  showed  that  the 
optimal  N-level  quantizer  is  identical  to  the  optimal  N-step  approximation  to  the  optimum  mean 
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squared-error  filter  for  the  source.  In  general,  the  quantizer  was  viewed  as  merely  an 
N-valued  zero-memory  filter.  Kurtenbach  and  Wintz  (88]  determined  optimum  and 
optimum-uniform  quantizer  structures  under  the  mean  squared-error  measure  for  the 
system  that  transmits  digital  data  over  a noisy  channel.  Their  proposed  design  procedures 
required  the  channel  transition  probabilities  to  be  related  to  the  bit  error  rate.  Gordan 
[83]  very  recently  presented  an  engineering-oriented  discussion  on  noise  effects  in  analog- 
to-digital  converters.  This  was  similar  to  Myers’  work.  Morris  [40,96]  investigated  the 
performance  of  an  arbitrary  quantizer  for  discrete-time,  memoryless,  noise-corrupted 
signal  sources  with  a given  amplitude  bound.  A functional  analysis  approach  was  used  to 
derive  the  results  on  worst-case  corrupting  noise  for  arbitrary  sources  and  quantizers.  His 
results  corroborated  the  findings  of  Myers. 

Estimation  and  Control  with  Quantized  Data 

After  these  few  results  were  obtained  from  the  study  of  the  optimal  quantization 
of  noise-corrupted  signal  sources,  several  researchers  investigated  the  problem  of  signal 
estimation  given  the  quantized  data.  This  is  a difficult  problem  because  the  conditional 
density  functions  are  in  general  complex.  Meier  et  al.  [93]  considered  the  effect  of 
quantization  of  the  linear  measurement  on  state  estimation  of  an  unforced  linear  dynamic 
system.  For  linear  estimation  and  certain  assumptions  on  the  density  function  at  each 
' time  step,  they  presented  a predictor-corrector  filter  of  the  Kalman  filter  form,  for  which 

the  measurement  noise  covariance  is  effectively  increased  by  an  additive  term  representing 
the  quantization  noise.  For  nonlinear  estimation  and  Gaussian  assumptions,  because  the 
resulting  conditional  density  functions  would  be  non-Gaussian,  approximate  methods 
yielding  tractable  recursive  estimates  were  presented.  These  approximate  methods  are 
based  on  the  most  probable  state  trajectory  estimate  or  the  most  probable  state  and  noise 
trajectory  estimate.  Curry  et  al.  [74]  and  Curry  [75]  presented  a general  method  of 
computing  statistics,  conditioned  on  quantized  measurements,  based  on  properties  of  the 
conditional  expectation.  On  application  to  Gaussian  discrete-time  linear  systems,  a Kalman 
filter  type  of  estimator  was  obtained.  For  nonlinear  estimators,  two  approximation 
methods  were  used  for  obtaining  tractable  estimators.  The  first  method  uses  a power 
series  expansion  of  the  Gaussian  density  function  to  approximate  the  mean  and  covariance 
of  the  conditional  density  function  for  a one-stage  problem.  The  second  method  uses  a 
Gaussian  fit  algorithm  on  the  conditional  density  function  at  each  stage,  which  yields  a 
Kalman  filter  type  of  estimator.  Clements  [71]  and  Clements  and  Haddad  [72]  determined 
expressions  for  the  conditional  density  functions  from  which  approximate  conditional  mode 
and  mean  estimates  can  be  determined.  They  treated  nonlinear  systems  in  this  context. 

By  using  Taylor  series  expansions  and  making  some  assumptions  on  the  system  equations, 
predictor-corrector  filter  equations  nae  obtained  for  the  estimators.  These  equations 
reduced  to  the  Kalman  filter  equations  for  linear  systems  and  no  quantization. 

References  71  and  72  are  more  general  than  the  others  because  they  treat  the  state 
estimation  of  fairly  general  nonlinear  systems.  However,  Curry’s  book  [75]  appears  to 
be  the  only  one  devoted  entirely  to  estimation  and  control  of  systems  with  quantized 
measurements.  It  also  contains  an  extensive  bibliography. 
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SUMMARY  AND  CONCLUSION 

Significant  research  results  relating  to  encoding  information  sources  have  been  high- 
lighted in  this  survey.  Particular  attention  has  been  given  to  the  application  of  quantiza- 
tion and  its  importance  in  source  encoding  in  a fidelity  criterion  context. 

Since  source  encoding  with  a fidelity  criterion  (i.e.,  rate-distortion  theory)  is  a 
relatively  new  area  of  research  activity,  the  literature  is  largely  theoretical.  Most  research 
has  been  confined  to  determining  source  encoding  schemes  and  rate-distortion  functions 
for  independent,  stationary,  Gaussian  sources  with  respect  to  mean  weighted-squared-error 
distortion  measures.  Some  theoretical  results  have  been  obtained  also  for  certain  deviations 
from  this  model,  most  notably  for  the  minimum  rates  of  Gaussian  sources  without  the 
independence  or  ergodic  assumptions  and  for  the  performance  of  specific  coding  or  quan- 
tization schemes  with  respect  to  the  standard  models.  Although  the  earliest  results  were 
generally  lacking  in  practicality  or  implementability,  with  the  exception  of  quantization, 
they  did  provide  insight  and  an  understanding  of  the  problem. 

There  are  still  problems  that  are  unsolved  or  in  early  stages  of  development.  These 
include  development  of  adaptive  and/or  robust  practical  source  encoders  for  classes  of 
general  sources,  sources  with  memory,  nonstationary  sources,  and  sources  with  vector 
outputs;  development  of  adaptive  and/or  robust  quantizers  for  these  same  types  of  sources; 
application  of  more  subjective  distortion  measures  or  fidelity  criteria  for  sources  with 
memory  and  symbol  context  dependencies,  such  as  in  speech  or  visual  images;  and  deter- 
mining the  rate-distortion  functions  for  these  sources  and  distortion  measures.  Successful 
investigation  of  these  areas  would  considerably  advance  the  use  of  this  theory  in  telecom- 
munication systems  design. 
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